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ABSTRACT 


In  order  to  enhance  the  usability  of  the  CyberCIEGE  to  assess  the  student’s  learning 
experience  and  achievement  of  the  scenario  objectives.  This  thesis  investigated  how  to 
improve  the  current  student  assessment  module,  report  generation  process,  report  fonnat 
and  also  integrating  of  CyberCIEGE  with  Naval  Postgraduate  School’s  (NPS)  Learning 
Management  System  (LMS).  Based  on  the  researched,  enhancements  such  as  additional 
summary  view  on  Event  Log  Analyzer  with  game  selection  features,  report  generation 
feature  on  the  Campaign  Analyzer,  various  levels  of  reports  and  the  process  of  linking  to 
the  Collaborative  Learning  Environment  (CLE)  was  implemented. 
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I.  INTRODUCTION 


A.  THESIS  STATEMENT 

The  purpose  of  this  thesis  is  to  create  a  Learning  Management  Platform  (LMP) 
for  CyberCIEGE.  The  LMP  will  provide  the  instructor  with  tools  and  an  environment  to 
better  analyze  the  student  learning  experience  and  achievement  of  the  scenario  objectives. 
This  would  include  investigating  how  to  improve  the  current  student  assessment  module, 
report  generation  process,  report  fonnat  and  also  integrating  of  CyberCIEGE  with  Naval 
Postgraduate  School’s  (NPS)  Learning  Management  System  (LMS). 

In  the  process  of  creating  the  CyberCIEGE’s  LMP,  the  following  questions  will 
be  answered: 

•  What  is  the  required  information  in  each  game  scenario  such  that  a 
meaningful  assessment  on  students’  understanding  of  the  course  materials 
can  be  carried  out? 

•  How  can  the  student  logs  be  better  documented  and  visualized  on  the  LMP 
so  that  the  assessment  of  objectives  for  respective  scenarios  can  be  clearly 
seen? 

•  How  best  can  the  CyberCIEGE’s  LMP  reports  be  integrated  into  Naval 
Postgraduate  School’s  LMS  to  enhance  the  teaching  experience? 

B.  RELATED  WORK 

CyberCIEGE’s  analysis  module  was  designed  by  Tiat  Leng,  Teo  as  part  of  his 
thesis  research  on  “Scenario  Selection  and  Student  Assessment  Module  for 
CyberCIEGE”  in  2003.  In  his  thesis,  he  highlighted  the  need  for  an  analysis  module  for 
training  system  so  that  it  would  allow  the  proper  conduct  and  review  of  the  training 
session  with  the  students.  However,  due  to  the  wide  scope  of  his  thesis,  he  only  managed 
to  implement  part  of  the  analysis  module,  which  will  be  described  in  detail  in  Chapter  II. 
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C.  CHAPTER  LAYOUT 

The  scope  of  this  thesis  is  to  enhance  the  current  analysis  module  in  order  to 
allow  a  more  efficient  and  effective  assessment  to  be  conducted  by  the  various  instructors 
of  courses  that  incorporate  CyberCIEGE  scenarios.  The  thesis  chapters  are  laid  out  in  the 
following  order: 

Chapter  I  Introduction 

This  chapter  highlights  the  thesis  statement  and  related  work  in  the  research  topic. 
It  also  gives  an  overview  of  the  thesis  layout. 

Chapter  II  Background 

This  chapter  highlights  some  of  the  key  comparisons  with  other  educational 
games’  assessment  modules.  Then  it  covers  the  background  infonnation  for  the  reader 
with  an  overview  of  the  current  assessment  module  and  the  report  generation  process. 

Chapter  III  Requirement  Analysis 

This  chapter  discusses  the  need  to  identify  the  various  objectives  of  the  designs  of 
individual  CyberCIEGE  scenarios.  It  also  looked  into  the  impetus  to  further  enhance  the 
current  assessment  module  and  what  level  of  detail  would  be  required  by  the  various 
course  instructors. 

Chapter  IV  Development  Strategies  and  Concepts 

This  chapter  describes  in  detail  the  implementation  of  the  improvement  to  the  log 
analysis  and  how  various  levels  of  reports  can  be  generated  by  filtering  the  required 
information  from  the  logs. 
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Chapter  V  Implementation  and  Testing 


This  chapter  discusses  the  implementation  methodologies  and  the  test  objectives 
to  ensure  that  all  requirements  to  enhance  the  instructor’s  module  have  been  implemented 
correctly. 

Chapter  VI  Conclusions  and  Recommendations 

This  chapter  provides  recommendations  and  suggestion  for  future  development  of 
the  CyberCIEGE’s  LMP  and  concludes  the  thesis. 
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II.  BACKGROUND 


A.  ANALYSIS  MODULE  OF  AN  EDUCATIONAL  GAME 

Education  through  games  allows  students  to  apply  what  was  taught  during  lessons 
in  a  simulated  real  life  scenario.  The  general  concept  of  educational  game  requires 
students  to  complete  certain  tasks  and  apply  the  acquired  knowledge  in  resolving  the 
issues  that  were  played  out  in  the  game  [1],  Many  serious  games  [2]  were  developed  over 
the  years  for  the  various  industries  e.g.,  Virtual  Battlespace  System  (VBS)  game  series 
[3]  were  developed  for  the  defense  industries,  History  of  Biology  was  developed  for  the 
bio-science  industry,  CyberCIEGE  was  developed  for  the  IA  committee,  X-Plane  for  the 
aviation  industry  etc.  However,  most  of  these  games  were  developed  to  create  some  level 
of  awareness  on  the  particular  topic  for  the  student  and  therefore  the  analysis  module 
mainly  comprised  a  simple  playback  function.  The  games  that  have  a  more 
comprehensive  analysis  module  were  actually  VBS  game  series  and  CyberCIEGE. 

When  VBS  first  rolled  out  its  initial  game  mission  in  2004  [4],  the  analysis 
module  or  After  Action  Review  (AAR)  module  features  were  limited.  It  only  comprised  a 
summary  of  the  mission  accomplishment  status.  When  this  game  was  adopted  by  the  US 
Army  and  US  Marine  Crops  as  the  basis  for  their  indoor  tactical  training  simulator,  it  was 
realized  that  a  more  comprehensive  AAR  module  need  to  be  incorporated  into  the  system 
to  reinforce  some  learning  objectives  [4]. 

Therefore,  in  the  recent  series  of  VBS2  missions,  a  more  robust  AAR  system  was 
integrated  into  the  VBS2  modules  allowing  the  perfonnance  of  the  crew  and  their 
equipment  e.g.,  coverage  of  the  weapon  systems,  field  of  view  of  the  night  vision 
devices,  to  be  shown  during  review.  The  AAR  module  adopted  the  U.S.  military  AAR 
framework  [5],  which  aims  achieved  the  following: 

•  To  identify  strengths  to  be  maintained  and  built  upon 

•  To  identify  potential  areas  for  further  improvement 

•  To  recommend  necessary  follow  up  actions 
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The  AAR  module  basically  collects  all  the  game  information  e.g. ,  movement,  hit, 
fire  weapon,  killed,  getting  in  and  out  of  actions,  voice  communication,  etc.  and  recreates 
the  scenario.  The  replay  scenario  can  be  viewed  from  any  player’s  perspective  in  two-  or 
three-  dimensional  view.  All  transcriptions  that  were  created  during  the  game,  reports  and 
forms,  were  also  embedded  in  the  AAR  file,  which  can  be  viewed  during  the  playback. 
However,  the  game  system  does  not  provide  a  summary  review  of  the  game  played. 

B.  CYBERCIEGE 

CyberCIEGE  is  an  educational  game  system  designed  with  the  intention  of 
educating  personnel  in  an  academic  setting.  The  game  exposes  the  player  to  realistic 
cyber  threats  that  a  network  administrator  might  face  in  real-life  situations.  The  player 
will  have  to  decide  on  the  most  efficient  and  effective  measures  to  prevent  the  loss  of  any 
confidentiality,  integrity  and  availability  of  a  fictional  company’s  critical  resources. 

Ever  since  the  rollout  of  this  game,  hundreds  of  educational  institutions 
worldwide  have  used  it  to  enhance  the  delivering  of  network  security  education  [6]. 
CyberCIEGE  is  built  with  various  campaigns  to  include  the  wide  range  of  network 
related  scenarios.  Details  of  the  CyberCIEGE  framework  will  be  discussed  in  Chapter  III. 
To  date,  improvements  to  the  game  system  and  scenarios  capturing  new  threats  and 
security  initiatives  have  subsequently  been  created  to  make  the  game  more 
comprehensive  and  realistic.  However,  the  development  of  the  analysis  module  was 
much  slower  as  compared  to  the  scenarios  development.  Therefore,  this  thesis  explores 
how  the  current  analysis  module  can  be  improved  to  further  enhance  the  teaching 
experience  using  CyberCIEGE. 

C.  CURRENT  ANALYSIS  MODULE 

The  current  analysis  module  for  CyberCIEGE,  the  “Campaign  Analyzer,”  was 
initially  developed  in  2003  [7],  [8].  The  Campaign  Analyzer  was  designed  to  provide  the 
instructors  with  a  tool  to  reconstruct  the  key  events  of  the  game  so  that  a  student’s 
progress  and  decisions  can  be  reviewed. 
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The  Campaign  Analyzer  basically  allows  an  instructor  to  select  a  specific 
campaign  and  scenario.  It  will  then  search  for  the  available  student  logs  files  that  contain 
the  game  progress  of  the  scenarios.  The  Campaign  Analyzer  will  then  summarize  the  raw 
logs  data  and  package  them  into  more  comprehensible  form,  as  shown  in  Figure  1. 


Figure  1.  Campaign  Analyzer. 


The  Campaign  Analyzer  is  divided  into  three  main  windows,  Campaign  window, 
Scenario  window  and  the  Summary  window,  from  the  top  to  bottom,  respectively.  The 
Campaign  window  encompasses  the  functions  for  the  instructor  to  select  the  campaign 
and  scenarios  within  the  campaign  to  be  viewed.  The  description  window  in  the 
Campaign  window  gave  an  overview  of  what  the  campaign  is  about. 

The  middle  panel  includes  the  Scenario  window,  which  gives  the  name  and 
description  of  the  selected  scenario.  It  also  allows  the  instructor  to  play  the  scenario  by 
clicking  the  “Play”  button. 
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The  last  panel  on  the  bottom  presents  the  summary  of  all  the  available  logs 
collected  for  that  selected  scenario.  The  summary  table  provides  key  information,  e.g., 
the  status,  the  time  spent,  the  number  of  times  played  for  that  scenario.  This  information 
would  allow  the  instructors  to  know  the  progress  of  his  students  and  make  necessary 
assessments. 

If  the  instructor  would  like  to  further  understand  what  happened  to  a  particular 
student’s  game,  he  only  need  to  select  the  student’s  name  and  click  on  the  “View  Log.” 
Once  the  “View  Log”  is  activated,  an  Event  Log  Analyzer  window  will  be  displayed  as 
shown  in  Figure  2. 


Figure  2.  Event  Fog  Analyzer. 
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The  Event  Log  Analyzer  displays  the  details  of  the  game  played  ranging  from  the 
game  engine  triggered  event  to  the  player  actions.  The  instructors  would  be  able  to  select 
to  number  of  games  to  be  viewed  for  the  particular  student.  This  allows  the  instructor  to 
step  through  what  the  student  has  encountered  and  look  into  what  actions  were  carried 
out  during  the  game  play.  If  a  game  was  saved  at  a  certain  stage,  the  instructor  is  able  to 
replay  the  game  to  understand  the  student’s  situation  better.  The  Event  Log  Analyzer 
also  provides  an  Event  type  filter  for  the  instructor  to  filter  out  any  information  which  is 
not  critical  for  his  analysis.  This  function  would  be  handy  if  the  instructor  knows  exactly 
what  type  of  information  he  wants  to  search  for. 

D.  REPORT  GENERATION  PROCESS 

The  initial  analysis  module  did  not  have  a  report  generation  function.  Therefore, 
in  order  to  overcome  that  limitation,  SQL  database  and  programs  were  developed  to 
generate  a  simple  report  for  the  instructors,  as  shown  in  Figure  3.  The  programs  basically 
extract  the  required  summary  data  using  the  Campaign  Analyzer  modules  and  place  it 
into  the  database. 
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|  WON (34) 

WON (12 9 

WON (37) 

mik 

i 

WON (78) 

|  WON (65) 

(73) 

WON (27) 

bran 

i 

WON (27) 

|  WON (20) 

WON (297 

WON (73) 

kf  e 

i 

WON (24) 

|  WON (23) 

(67) 

WON (32) 

Figure  3 .  Sample  of  Summary  Report 
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In  this  summary  report,  the  top  row  shows  the  lab  number  and  the  extreme  left 
column  contains  the  user  names.  The  report  is  comprised  of  the  status  of  each  game  with 
the  minutes  spent  in  brackets.  The  “ — ”  indicates  that  the  student  did  not  attempt  the  lab. 
This  summary  report  only  provides  the  most  basic  set  of  information  for  the  instructors. 

On  top  of  that,  in  order  for  the  script  to  work,  the  user  must  be  able  to  manually 
edit  the  following  parameters  within  the  script  file: 

•  Set  classpath — The  classpath  directs  the  script  to  pull  the  relevant  java 
class  so  that  required  information  can  be  generated. 

•  Set  student  list — The  student  list  contains  the  name  of  a  particular  class.  It 
will  direct  the  script  to  pull  out  infonnation  from  the  relevant  student’s  log 
file. 

•  Set  game — Setting  the  game  would  direct  the  script  to  pull  the  necessary 
information  from  the  respective  campaign  and  scenario. 

•  Set  output  path — This  gave  the  script  an  output  file  name  for  the  summary 
report. 

This  approach  to  generating  the  summary  report  is  not  user  friendly,  as  not  all  the 
instructors  would  know  how  the  script  can  be  edited  and  the  correct  syntax  of  the  file 
format  to  be  included  for  the  script.  Also  the  report  only  provides  sufficient  for  the 
instructor  to  assess  which  student  has  completed  the  lab  assignment.  In  the  event  where 
the  instructors  want  to  look  into  the  details  for  individual  student,  he  would  have  to  view 
it  from  the  Campaign  Analyzer  module  with  the  student’s  log  file.  Thus,  an  improved 
summary  report  with  sufficient  details  captured  is  required  for  better  assessment  by  the 
instructors. 

E.  SUMMARY 

As  we  move  into  a  new  era  of  education  through  digital  games,  the  analysis 
module  is  one  of  the  key  success  components  to  ensure  that  the  required  knowledge  was 
correctly  and  clearly  transmitted  through  the  game  play.  Likewise,  a  clear  and  concise 
summary  report  is  also  handy  when  the  game  is  deployed  in  an  academic  setting  where 
various  classes  of  students  will  play  the  game  in  the  lab. 
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III.  REQUIREMENT  ANALYSIS 


This  chapter  discusses  three  key  approaches,  game  design,  summary  report  design 
and  integration  with  the  Sakai  CLE,  to  enhance  the  analysis  of  the  student  learning 
experience  and  achievement  of  the  scenario  objectives.  In  order  to  have  a  meaningful 
student  assessment,  the  planning  starts  from  the  game  design  to  ensure  that  all  the 
intended  knowledge  and  information  are  correctly  embedded  into  the  game  play.  The  first 
section  will  review  the  scenario  design  framework  and  recommend  some  improvement  to 
it.  Following  that,  the  next  section  explores  how  the  key  information  can  be  captured  and 
presented  in  a  relevant  report  structure.  The  last  section  discusses  the  possibility  of 
integrating  the  CyberCIEGE  game  system  into  Sakai  Collaborative  Learning 
Environment  (CLE). 

A.  GAME  DESIGN 

This  section  reviews  the  structure  and  context  of  game  design  for  the  purpose  of 
exploring  ways  of  assessing  whether  learning  objectives  are  met.  Objectives 
identification  should  be  conducted  before  the  commencement  of  the  scenario  design,  as 
the  objectives  will  eventually  shape  the  design  of  the  scenario. 

1.  Game  Framework 

The  structural  framework  of  a  CyberCIEGE’s  framework  is  shown  in  Figure  4. 

below: 
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Phase 


Figure  4.  CyberCIEGE  Game  Framework 


The  Campaign  is  the  overarching  title  for  the  various  scenarios  that  are  to  be 
grouped  together.  The  campaign  will  determine  the  general  classification  of  the  types  of 
attacks  or  scenarios  that  will  be  built  within  its  scope.  The  scenario  design  is  the 
development  step  where  the  designer  has  to  carefully  plan  the  storyline  and  other  game 
details  like  the  characters,  equipment,  environment  layout,  etc.  Depending  on  the  layout 
of  the  scenario,  the  scenario  designer  can  then  further  break  the  scenario  into  various 
phases  so  that  the  player  can  progressively  learn  and  move  through  the  scenario.  This 
method  of  breaking  the  scenario  into  phases  will  ensure  that  the  student  understands  and 
able  to  apply  the  knowledge  before  going  into  the  next  phase.  This  method  can  also 
provide  the  instructor  with  an  assessment  of  the  student’s  understanding  and  an  ability  to 
provide  necessary  coaching  when  required.  An  example  of  how  the  framework  can  be 
laid  out  in  the  Introductory  VPNs  (Virtual  Private  Network)  is  as  follows  [9]: 
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CAMPAIGN 

SCENARIO 

PHASE  DESCRIPTION 

DESCRIPTION 

DESCRIPTION 

Encryption 

Introductory  VPNs 

Phase  1 

-  This  campaign  illustrates 

-  Help  Lisa  communicate 

-  Secure  Harry  and  Lisa’s 

the  use  of  cryptography  to 

securely  over  the  Internet. 

communication  over  the 

protect  communications 

Harry  and  Lisa  collaborate 

internet 

from  unauthorized 

on  a  critical  marketing 

Phase  2 

disclosure  and  modification. 

This  campaign  also  includes 

techniques  for  managing  the 

roadmap  from  different 

parts  of  the  city.  Lisa 

accesses  the  roadmap  via 

-  Harry  and  Lisa  would  like 

to  access  the  web 

“identity”  of  data,  i.e., 

the  Internet,  but  the  boss 

Phase  3 

authenticating  the  origin  of 

believes  that  may  be  risky. 

-  Protect  moderate  value 

data.  Topics  include  use  of 

In  particular  he  wants  to 

assets 

Virtual  Private  Networks 

(VPNs),  email  encryption 

and  SSL. 

make  sure  that  only  Harry 

and  Lisa  can  modify  the 

roadmap.  Your  job  is  to  help 

Lisa  secure  her  Internet 

communication. 

Phase  4 

-  Protect  high  value  assets 

Table  1.  Example  of  VPN  Framework  Layout 

2.  Scenario  Design 

The  construction  of  the  game  occurs  during  the  scenario  design.  Therefore,  the 
objectives  must  be  clearly  spelled  out  so  that  the  rest  of  the  supporting  elements  can  be 
built  to  support  the  objectives.  In  the  scenario  design  there  are  two  key  objectives  to  be 
identified;  scenario  objectives  and  educational  objectives. 

The  scenario  objectives  highlight  what  the  scenario  is  designed  for.  It  lays  out  the 
scope  of  the  game  and  also  the  general  path  for  how  the  scenario  should  be  built.  A 
review  of  previous  thesis  that  included  CyberCIEGE  scenario  building  [10]  ,[11],  [12], 
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[13],  [14],  revealed  that  all  the  authors  did  managed  to  define  the  objectives  of  the 
scenario,  thus  I  will  not  further  elaborate  that  in  this  thesis.  Using  the  Introductory  VPNs 
scenario  as  an  example,  the  scenario  objective  could  be  defined  as  “To  create  an 
environment  for  the  user  to  practice  the  deployment  of  a  VPN  for  a  corporate  network.” 

CyberCIEGE  is  a  gaming  tool  that  is  designed  to  educate  players  about  computer 
and  network  security,  therefore,  the  educational  objectives  should  be  carefully  crafted  in 
order  to  outline  the  goals.  The  scenario  designer  sets  educational  objectives  to  ensure  that 
the  players  will  acquire  the  correct  and  intended  information  during  the  conduct  of  the 
game.  This  set  of  objectives  will  help  the  scenario  developer  refine  the  game  storyboard 
and  eventually  these  goals  could  be  used  to  formulate  the  lab  learning  objectives  as  well. 
Some  examples  of  the  educational  objectives  for  the  Introductory  VPN  scenario  are: 

•  To  illustrate  that  unprotected  internet  connection  can  be  hijacked  by 
attackers. 

•  Risks  of  VPN  gateways  that  allow  both  protected  and  unprotected  traffic. 

•  Limitation  of  link  encryp tors. 

•  Configuration  of  VPN  gateways  to  constrain  their  sources  and 
destinations. 

With  the  educational  objectives  being  defined,  the  scenario  developer  will  then 
design  the  game  with  the  required  supporting  elements  to  achieve  the  objectives  set  forth. 
Likewise  the  in-game  question  function  could  be  used  to  achieve  the  objectives  in  the 
event  when  an  objective  is  not  obvious  or  not  achievable  in  the  scenario  play.  This 
function  allows  the  developer  to  create  a  pop-up  dialogue  box  to  seek  input  from  the 
player  in  a  form  of  multiple-choice  questions.  It  is  also  configurable  such  that  the 
dialogue  will  be  looped  continuously  till  the  correct  answer  is  chosen.  The  selection  of 
the  result  can  also  be  captured  in  the  log  file  for  further  assessment  in  the  later  stage. 

Therefore,  with  all  the  objectives  and  check-points  being  built  into  the  game 
scenario,  the  instructors  will  be  able  to  design  the  lab  training  session  effectively  and 
synchronizing  it  with  the  materials  taught  during  lessons.  Likewise,  when  a  student  is 
stuck  at  a  certain  phase,  the  instructor  is  able  to  correlate  the  issues  depending  on  the 

stage  they  are  at.  Similarly,  the  instructors  can  also  look  at  the  answers  to  the  quiz 
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questions  and  understand  what  the  common  wrong  choices  that  students  made.  Other 
than  using  the  Campaign  Analyzer  to  look  at  all  the  student’s  progress  as  mentioned,  a 
summary  report  capturing  all  the  essential  information  would  allow  the  instructors  who 
are  not  familiar  with  the  Campaign  Analyzer  to  make  similar  assessment.  The  next 
section  will  cover  the  requirements  of  what  a  comprehensive  summary  report  should 
have. 

B.  SUMMARY  REPORT  ENHANCEMENT 

As  mentioned  in  the  previous  chapter,  CyberCIEGE  has  been  used  by  educators  at 
hundreds  of  educational  institutions.  Among  these,  educators  fall  into  two  broad 
categories;  advanced  users  and  basic  users.  The  advanced  users  are  those  who  know 
every  component  of  the  game  engine  ranging  from  the  game  scenarios  to  the  assessment 
modules.  These  users  are  able  to  identify  the  specific  scenarios  and  use  them  to  enhance 
the  conduct  of  network  security  lessons  or  lab  sessions.  Upon  completion  of  the  training 
sessions  by  the  students,  these  educators  will  be  able  to  assess  and  track  student 
achievement  through  use  of  the  Campaign  Analyzer  and  Event  Log  Analyzer. 

The  second  class  of  educators  is  the  basic  user.  These  users  know  the  basic 
scenario  set-up  and  are  able  to  design  and  conduct  lessons  or  lab  sessions  with  the 
available  scenarios.  However,  these  users  are  not  necessarily  familiar  with  the  assessment 
module  and  they  are  dependent  on  a  CyberCIEGE  administrator  to  generate  the  required 
reports  or  they  merely  use  it  with  alternate  methods  of  assessment  or  analysis  of  the 
student’s  performance.  Therefore,  in  order  to  cater  to  the  needs  of  various  users,  a 
smoother  way  of  generating  the  summary  report  and  a  extending  the  report  to  include 
information  necessary  for  assessment  are  required. 

In  order  to  create  a  seamless  report  generation  process,  the  report  generating 
function  should  be  integrated  into  the  Campaign  Analyzer  Module.  This  would  allow  the 
instructor  to  have  his  initial  assessment  using  the  Campaign  Analyzer  and  in  the  event 
that  a  summary  report  is  required,  the  report  could  be  generated  via  clicking  of  button  in 
the  Campaign  Analyzer.  The  function  should  allow  the  user  to  select  the  result  from  the 
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relevant  scenarios  and  extract  the  necessary  infonnation.  It  should  also  have  the 
flexibility  to  filter  the  selected  students’  log  files  for  the  class. 

The  next  step  is  to  look  at  the  requirements  for  a  good  summary  report.  As  shown 
in  Chapter  II,  the  current  summary  report  only  contains  the  basic  infonnation  of  status 
and  time  spent  on  that  particular  lab.  The  information  only  helps  instructor  know  who  has 
attempted  and  completed  the  assigned  lab.  Therefore,  a  second  level  of  report  that 
encompasses  more  details  should  also  be  generated  to  allow  a  more  effective  assessment. 

For  the  first  level  summary  report,  a  proper  listing  of  the  status  could  be 
implemented  as  compared  to  the  currently  the  report  which  only  shows  the  “Won”  status. 
When  a  player  lost  or  quit  the  game,  only  the  total  time  spent  is  shown  in  the  status  area. 
Likewise,  the  total  time  spent  shown  in  the  status  might  not  be  enough  to  assess  the 
student’s  achievement;  therefore,  additional  information  like  the  number  of  times  played 
could  be  included,  as  shown  in  Figure  5. 


Student  Name 

Lab  1 

Lab  2 

Lab  3 

Lab  4 

StudentOOl 

WON(10/2) 

WON(17/3) 

NOT  PLAYED 

WON(16/2) 

Student002 

LOST(20/2) 

WON  (14/2) 

LOST(  725/8) 

NOT  PLAYED 

Student003 

WON(23/3) 

WON(9/l) 

WON(76/5) 

WON(18/2) 

Student004 

WON(22/2) 

WON  (18/2) 

WON(16/3) 

NOT  PLAYED 

Student005 

WON(14/l) 

WON(23/2) 

WON(20/2) 

WON(10/1) 

Student006 

WON(54/4) 

WON(18/2) 

WON(2l/l) 

WON(33/3) 

Student007 

LOST(17/2) 

LOST(50/5) 

WON(82/6) 

NOT  PLAYED 

Student008 

WON(15/l) 

WON(72/6) 

LOST(23/2) 

WON(19/3) 

Student009 

WON(27/3) 

WON(22/2) 

WON(10/1) 

NOT  PLAYED 

StudentOlO 

WON(36/5) 

WON(43/5) 

WON(33/3) 

WON(27/2) 

Figure  5.  Improved  Level  1  Summary  Report. 


As  for  the  second  level  of  report,  it  could  present  a  framework  similar  to  the 
Campaign  Analyzer.  The  details  can  also  be  further  broken  down  into  the  attempts  for 
each  student  in  a  student  view  table  (Figure  6)  or  a  summary  of  the  lab  attempts  in  a  lab 
view  table  (Figure  7)  with  all  the  student  attempt  information.  This  will  allow  the 
instructor  to  have  a  good  view  of  both  the  class  perfonnance  and  individual  achievement 
for  the  scenario  just  from  the  summary  report. 
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STUDENT001 

LAB  1 

ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUIT) 

POINT  OF  LOST 

1 

Ol-Jul-11 

3 

LOST 

Phase  1 

2 

03-Jul-ll 

7 

WON 

— 

LAB  2 

ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUIT) 

POINT  OF  LOST 

1 

22-Jul-ll 

2 

QUIT 

— 

2 

22-Jul-ll 

5 

LOST 

Phase  2 

3 

22-Jul-ll 

10 

WON 

— 

LAB  3 

ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUIT) 

POINT  OF  LOST 

LAB  4 

ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUIT) 

POINT  OF  LOST 

1 

Ol-Aug-11 

10 

WON 

— 

2 

Ol-Aug-11 

6 

WON 

— 

Figure  6.  Student  View  Summary  Table 


LAB  1 

STUDENT  001 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUm 

POINT  OF  LOST 

1 

Ol-Jul-11 

3 

LOST 

Phase  1 

2 

03-Jul-11 

7 

WON 

— - 

STUPENT002 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUm 

POINT  OF  LOST 

1 

02- Jul-11 

12 

LOST 

Phase  2 

2 

02-Jul-11 

8 

LOST 

Phase  2 

STUDENT  003 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUm 

POINT  OF  LOST 

1 

Ol-Jul-11 

7 

LOST 

Phase  2 

2 

Ol-Jul-11 

10 

WON 

3 

02- Jul-11 

6 

WON 

.... 

STUDENT  004 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUrT) 

POINT  OF  LOST 

1 

Ol-Jul-11 

13 

WON 

.... 

2 

Ol-Jul-11 

3 

WON 

— - 

STUDENT005 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST /QUIT) 

POINT  OF  LOST 

1 

Ol-Jul-11 

U 

WON 

.... 

Figure  7.  Lab  View  Summary  Table 


Analytical  graphs  and  charts  could  also  be  implemented  in  the  summary  report  to 
enhance  the  assessment.  A  few  variants  of  the  charts  that  could  be  implemented  are  as 
follows: 

•  Time  Spent  Analysis  Chart  (Figure  8) — Time  spent  on  each  scenario  could 
be  used  as  a  proxy  to  identify  which  student  has  attempted  the  lab. 
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However,  it  cannot  be  used  as  a  good  assessment  tool  for  which  student  is 
better  as  some  students  might  be  exploring  the  other  parts  of  the  scenario. 


Time  Spent  Chart 


Student  Student  Student  Student  Student 
1  2  3  4  5 

Student  Name 


■  Lab  1 

■  Lab  2 

■  Lab  3 


Figure  8.  Time  Spent  Analysis  Chart 


•  Attempt  Analysis  Chart  (Figure  9) — The  attempt  chart  is  a  better  analysis 
chart  compared  to  the  Time  Spent  Chart.  This  chart  enables  the  instructor 
to  assess  who  has  attempted  and  on  top  of  that  he  is  also  able  to  know 
which  student  has  completed  the  lab  successfully.  Such  a  chart  might  aid 
conclusions  like  whether  this  is  an  easy  lab  or  whether  the  students  have 
enough  knowledge  to  prior  to  attempting  the  lab.  This  would  also  allow 
the  instructors  to  review  his  teaching  materials  so  that  the  students  are  able 
to  handle  the  problem  set  in  the  game  scenario. 
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Attempt  Chart  For  Lab  1 


■  Won 

■  Lost 


Student  Student  Student  Student  Student 
1  2  3  4  5 

Student  Name 

Figure  9.  Attempt  Analysis  Chart 

•  Scenario  Analysis  Chart  (Figure  10) — This  chart  would  provide  the 
instructor  with  information  on  which  part  of  the  game  scenario  is  too 
difficult  for  the  student  to  handle.  From  the  analysis,  the  instructor  would 
be  able  to  decide  whether  to  cover  more  material  on  that  particular  subject 
or  provide  more  instructions  for  that  particular  scenario. 


Scenario  Failure  Chart 


Lab  1  Lab  2  Lab  3 
Lab  Number 


Lab  4 


■  Phase  1 

■  Phase  2 

■  Phase  3 

■  P  ha  se  4 


Figure  10.  Scenario  Analysis  Chart 
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Once  the  reports  are  generated,  the  next  step  is  to  deliver  these  reports  to  the 
respective  instructors.  Prior  to  this  thesis  work,  the  NPS  approach  was  to  deliver  these 
summary  reports  via  hardcopy  or  through  e-mail.  The  next  section  will  discuss  how  the 
summary  report  could  be  delivered  to  the  instructors  through  a  CLE. 

C.  INTEGRATION  WITH  A  CLE 

Many  educational  organizations  utilize  CLE  platfonns  to  centralize  course 
materials  whereby  the  course  notes,  assignments  and  quizzes  are  conducted  within  the 
respective  course  site.  This  approach  has  made  the  management  of  courses  much  easier 
for  the  instructors.  NPS  is  one  of  over  350  educational  organizations  that  have  adopted 
the  Sakai  CLE  as  a  learning  management  system  [15].  Sakai  CLE  is  a  platfonn  that 
allows  academic  and  research  collaboration  and  it  is  built  on  open  pedagogy  and  open 
standard  [16]. 

In  those  situations  where  the  course  instructor  is  not  the  one  who  is  gathering  the 
logs  and  creating  the  reports,  it  would  be  useful  to  provide  those  instructors  with  report 
output  within  the  Sakai  environment.  To  more  fully  integrate  CyberCIEGE  into  Sakai,  we 
would  need  to  look  at  how  CyberCIEGE  can  adopt  the  Sharable  Content  Object 
Reference  Model  (SCROM)  standards  [17].  SCROM  is  a  set  of  technical  standards,  (for 
which  Sakai  claims  compliance),  that  specify  three  key  areas;  how  content  should  be 
packaged  and  described,  how  content  should  be  launched  and  how  students  can  navigate 
between  parts  of  the  course.  The  tools  that  manage  CyberCIEGE  scenario  packaging  and 
student  access  to  scenarios  could  be  adapted  to  operate  within  a  SCROM-compliant 
context,  however  that  is  beyond  the  scope  of  this  thesis,  but  it  will  be  addressed  further  in 
Chapter  VI  for  possible  future  work. 

For  the  purpose  of  this  thesis,  we  will  only  explore  how  to  integrate  the  summary 
report  generated  by  CyberCIEGE  into  the  Sakai  CLE.  The  details  of  that  implementation 
will  be  covered  in  the  next  chapter. 
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IV.  DEVELOPMENT  STRATEGIES  AND  CONCEPTS 


This  chapter  describes  the  development  strategy  and  concepts  in  developing  the 
solution  to  provide  a  more  efficient  CyberCIEGE  student  assessment  environment.  The 
development  strategy  is  to  divide  the  work  in  three  key  phases  that  will  be  discussed  in 
the  next  three  sections.  Each  of  these  phases  will  explain  the  selected  requirement  listed 
in  Chapter  III  and  how  these  requirements  were  implemented.  The  first  phase  will  look  at 
how  the  Event  Log  Analyzer  can  be  improved  to  facilitate  the  assessment  for  instructors 
who  are  using  the  module.  The  next  phase  will  discuss  what  infonnation  should  be 
captured  for  the  different  level  of  summary  reports  and  how  the  framework  of  the  report 
should  be  like.  The  last  phase  will  look  at  how  the  summary  report  can  be  integrated  into 
Sakai  CLE. 

A.  IMPROVEMENT  TO  THE  CURRENT  EVENT  LOG  ANALYZER 

As  described  in  Chapter  II,  the  Event  Log  Analyzer  window  will  be  activated 
when  the  user  selects  the  log  file  of  a  particular  student  and  clicks  on  the  view  log  button 
on  the  Campaign  Analyzer  window  as  shown  in  Figure  1 1 .  The  top  panel  of  the  resulting 
window  shows  the  event  list  of  the  student’s  game  log  sorted  by  actual  date  and  time.  The 
middle  panel  shows  the  details  of  the  event  whenever  an  event  is  selected  on  the  top 
panel.  The  bottom  panel  allows  the  user  to  filter  the  required  event  and  the  number  of 
games  to  be  shown  on  the  top  panel. 
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Figure  1 1 .  Event  Log  Analyzer  description. 

There  are  two  limitations  with  the  current  Log  Analyzer.  The  first  is  that  it  is  not 
able  to  show  only  a  single  game  session  except  the  last  one.  This  is  because  the  design  of 
the  module  only  allows  the  user  to  display  the  number  of  last  log  files  to  be  presented. 
For  example,  if  a  student  plays  a  particular  scenario  four  times,  and  if  the  instructor 
would  like  to  look  at  the  second  game  that  the  student  played,  the  module  will  only  allow 
the  instructor  to  load  the  last  three  games  such  that  the  instructor  has  to  ignore  the  other 
two  games  at  the  end  of  the  event  list.  The  other  issue  identified  is  that  both  the 
Campaign  Analyzer  and  the  Event  Log  Analyzer  do  not  provide  a  summary  of  all  the 
attempts  by  the  respective  student  in  that  scenario.  This  makes  the  assessment  difficult  as 
the  instructor  has  to  scan  through  the  full  event  list  for  every  student  in  order  to  know 
how  the  student  did  for  each  of  the  attempt. 

Therefore,  this  thesis  implemented  the  following  improvements  to  the  Event  Log 
Analyzer. 
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•  Include  a  summary  table  that  shows  all  the  student’s  attempts  on  the 
scenario. 

•  Display  only  the  selected  game  attempt  by  the  user. 

To  design  the  summary  table,  it  was  required  to  first  identify  the  information  that 
is  essential  for  the  instructor.  This  infonnation  should  provide  the  instructor  with 
sufficient  details  so  that  he  can  decide  whether  to  look  further  into  that  particular  game 
attempt.  The  identified  infonnation  is  as  follows: 

•  The  attempt  number  by  the  student. 

•  The  status  for  that  attempt  (Won,  Lost  or  Quit). 

•  The  amount  of  cunency  at  the  end  of  that  attempt. 

•  The  real  time  that  attempt  started. 

•  The  real  time  that  attempt  ended. 

•  The  number  of  minutes  played  for  that  attempt. 

In  order  to  display  the  selected  event,  the  best  option  is  to  allow  the  user  to  select 
the  game  straight  from  the  summary  table.  This  approach  allows  the  instructor  to  select 
the  game  attempt  directly  from  the  information  on  the  summary  table.  Therefore,  the 
summary  table  should  be  positioned  at  the  top  panel  of  the  Event  Log  Analyzer  so  that 
the  user  can  first  look  at  the  game  attempts  and  then,  upon  selection  of  the  particular 
attempt,  the  sequence  of  events  appear  on  the  second  panel  as  shown  in  Figure  12. 
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Figure  12.  Improved  Event  Log  Analyzer. 


B.  SUMMARY  REPORT 

The  summary  report  is  important  for  those  instructors  who  are  not  personally 
collecting  and  processing  the  logs  files.  The  key  concept  for  the  summary  report  is  to 
keep  it  clear  and  concise  while  providing  sufficient  information  for  student  assessments 
to  be  made.  With  that  in  mind,  if  the  summary  report  was  broken  down  into  two  levels. 
The  first  level  provides  the  basic  achievement  of  each  individual  student.  If  deemed 
required,  the  instructor  would  then  click  on  the  student  name  to  go  into  the  second  level 
of  detail  for  further  assessment. 

The  information  set  identified  for  level  1  is  as  listed. 

•  Student  name 

•  Lab  Status  to  show  the  state  of  progress  for  each  student.  (Won,  Lost  or 
Quit) 
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•  Total  time  spent 

•  Number  of  attempts  for  each  lab 

In  order  to  ensure  clarity  in  the  report,  necessary  terms  e.g.,  mins,  game,  etc.  are 
included  in  the  summary  table.  The  report  also  utilizes  color  shading  to  highlight  those 
who  are  yet  to  start  the  lab  and  also  those  who  have  lost  or  quit  the  game  to  enhance  the 
identification  of  possible  problem  areas,  as  shown  in  Figure  13. 


SUMMARY  REPORT  (  quarter  fall-2010  taurus  ) 


STUDENT  NAME 

LAB  1 

LAB  2 

LAB  3 

LAB  4 

StudentOl 

Won  (7  mins  1  game) 

Won  (32  mins  3  game) 

Won  (8  mins  3  game) 

Quit:GoPhaseOne -1  (11  mins/1  game) 

Student02 

Not  started 

Won  (10  mins  1  game) 

Won  (13  mins  3  game) 

Not  started 

StudentOS 

Not  started 

Won  (19  mins  2  game) 

Won  (16  mins  4  game) 

Not  started 

Student04 

Not  started 

Won  (27  mins  7  game) 

Quit  (21  mins/4  game) 

Not  started 

StudentOf 

Not  started 

Won  (27  mins  1  game) 

Won  (21  mins  1  game) 

Not  started 

Student06 

Not  started 

Won  (83  mins  5  game) 

Won  (28  mins  5  game) 

Not  started 

StudentOl 

Not  started 

Quit:goPhase3-2  (26  mins/2  game) 

Won  (IS  mins  2  game) 

Not  started 

StudentOS 

Not  started 

Won  (17  mins  1  game) 

Won  (12  mins  4  game) 

Not  started 

Figure  13.  Level  1  Summary  Report. 


The  level  2  summary  report  should  be  linked  seamlessly  from  the  level  1  report. 
A  link  function  was  built  to  hyperlink  to  the  level  2  report  when  the  user  clicks  on  the 
“Student  Name.”  The  level  2  report  shows  all  the  lab  sessions  and  details  of  the 
achievement  in  each  attempt  within  that  particular  lab  session.  For  example  if  the 
instructor  would  like  to  look  at  student  Student07  game  sessions,  he  will  click  on 
Cooper’s  name  and  it  would  lead  him  to  the  level  2  report,  as  shown  in  Figure  14. 
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Student07’s  Lab  Summary 
LAB  1 


ATTEMP  NOS 

STATUS 

CASH 

TIME  STARTED 

TIME  ENDED 

MINUTES  PLAYED 

1 

Not  Played 

— 

— 

— 

— 

LAB  2 


ATTEMP  NOS 

STATUS 

CASH 

TIME  STARTED 

TIME  ENDED 

MINUTES  PLAYED 

1 

Lost:goPhase2— 0 

-7569.0 

06  Dec  2010  23:08:26 

06  Dec  2010  23:22:12 

13 

2 

Quit:goPhase3-0 

966.0 

06  Dec  2010  23:22:38 

06  Dec  2010  23:36:18 

13 

LAB  3 


ATTEMP  NOS 

STATUS 

CASH 

TIME  STARTED 

TIME  ENDED 

MINUTES  PLAYED 

1 

Won 

3657.0 

20  Oct  2010  21:46:23 

20  Oct  2010  22:03:31 

17 

2 

Lost 

-1996060.0 

20  Oct  2010  22:04:10 

20  Oct  2010  22:05:15 

1 

LAB  4 


ATTEMP  NOS 

STATUS 

CASH 

TIME  STARTED 

TIME  ENDED 

MINUTES  PLAYED 

1 

Not  Played 

— 

— 

— 

— 

BACK  TO  TOP 


Figure  14.  Level  2  Summary  Report. 


From  the  report,  the  instructor  is  able  to  deduce  the  following: 

•  The  student  has  yet  to  start  his  lab  1  and  lab  4  sessions  (this  conclusion 
can  also  be  made  from  level  1  summary). 

•  The  student  has  attempted  lab  2  twice,  and  the  student  lost  the  game  on  the 
first  attempt.  The  student  quit  on  the  second  attempt  of  phase  3. 

•  The  student  has  attempted  lab  3  twice.  He  has  successfully  achieved  the 
game  objectives  on  the  first  attempt.  The  student  attempted  the  lab  again 
and  could  be  trying  out  a  different  configurations  or  choices  for  his  own 
learning  purposes. 

Now  that  both  the  level  1  and  2  summary  reports  have  been  described,  we  will 
look  at  how  the  summary  report  is  generated  and  what  type  of  file  format  the  reports 
should  be  in.  The  report  generation  function  was  built  into  the  Campaign  Analyzer 
module,  as  shown  in  Figure  14,  to  allow  the  user  to  generate  the  summary  reports  at  a 
click  of  the  button. 
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I4»|  Campaign  Analyzer  [  a  ||  HI  ||  22  | 


Campaign 


Name 

Scenario  List 

Starting  Scenarios 

Select 

Introduction  Scenario 

Description 

Physical  Security 

TirePly  Filter  Scenario 

Patches 

PCA 

These  scenarios  illustrate  a  variety  of  computer  and  network  security  challenges. 

The  first  scenario  includes  substantial  help  tips  to  familiarize  players  with  mechanics 
of  the  game.  Players  are  encouraged  to  make  "mistakes"  and  explore 
consequences  of  different  decisions.  These  scenarios  are  intended  to  prepare 
players  for  the  Encryption  and  Identity  Management  scenarios  that  cover  a  broader 

= 

Close 


Figure  15.  Campaign  Analyzer  with  Generate  Report  Function 


Upon  clicking  the  “Generate  Report”  button,  another  interface  appears  to  let  the 
user  select  the  required  input  and  output  fdes,  as  shown  in  Figure  16. 
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I4>l  Input  and  Output  Selector 


S3 


INPUT  AND  OUTPUT  SELECTOR 


Select  Lab  Mapping  File 

C:\Users\AdminNUS\Desktop\Thesis\CyberCIEGE\Ccse\labmap.txt 


Select 


Select  Student  Mapping  File 

C:\Users\AdminNUS\Desktop\Thesis\CyberCIEGE\Ccse\studentmap2.bct 


Select 


Select  Output  Path 

C:\Users\AdminNUS\Desktop\Thesis  Output 


Select 


Generate 


Close 


Figure  16.  Input  and  Output  Selector  Dialog  Window 


This  dialog  selects  two  input  mapping  fdes  and  one  output  path.  The  first  file 
identifies  how  the  scenario  and  lab  session  are  mapped  together.  The  information  from 
that  file  directs  the  system  to  pull  information  from  the  corresponding  scenarios  and 
package  them  into  the  respective  lab  sessions.  The  second  input  file  is  comprised  of  the 
class  description  and  the  name  of  students  who  belong  to  that  class.  The  system  will  use 
the  class  description  for  both  the  title  of  the  summary  report  and  the  output  file  name  for 
easy  identification  of  the  report.  The  name  of  the  students  will  allow  the  system  to 
identify  the  correct  student  log  files  to  be  pulled  out  for  the  summary  report.  This  step  is 
required  because  the  person  administering  the  logs  might  be  doing  so  for  more  than  one 
class. 

The  key  considerations  for  the  output  file  format  is  that  it  should  be  able  to  hold 
both  level  1  and  level  2  summary  report  in  one  single  file.  This  is  necessary  so  to  prevent 
overloading  the  instructors  with  too  many  files.  Having  that  in  mind,  two  options  were 
explored,  the  excel  file  fonnat  and  the  html  file  format.  The  excel  format  is  able  to  store 
the  various  reports  using  the  “tab”  function.  It  also  provides  a  good  platform  for  the 
instructors  to  conduct  further  analysis  with  graphs  and  charts  provided  in  the  excel 
program.  However,  for  this  format  to  be  usable,  the  instructor  would  need  to  have  Excel 
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program  installed  in  his  system.  This  pre-requisite  could  be  limiting  as  not  all  educational 
institutions  have  the  Excel  program.  Therefore,  the  html  format  was  chosen  as  the  report 
file  format  and  this  format  also  provides  easier  integration  to  Sakai  CLE. 

C.  INTEGRATION  WITH  SAKAI  CLE 

The  integration  was  made  easy  by  the  sharing  features  of  the  Sakai  CLE,  which 
employs  the  Web-based  Distributed  Authoring  and  Versioning  (WebDAV)  protocol  that 
facilitates  collaboration  in  Hypertext  Transfer  Protocol  (HTTP).  However,  during  the 
testing  of  the  WebDAV  interface  of  the  NPS  CLE,  a  WebDAV  client  command  line 
failed  to  connect  to  the  CLE.  Similarly,  a  popular  WebDAV  drag-and-drop  client  called 
WinSCP  was  unable  to  connect  to  the  CLE.  Therefore,  an  alternative  connection  strategy 
using  Window  7  drive  mapping  was  chosen. 

As  Window  7  has  a  built-in  WebDAV  client,  the  user  can  use  the  Windows 
network  drive  mapping  function  to  map  a  drive  using  the  unique  URL  created  for  each 
course  site  on  the  CLE.  During  the  mapping  of  the  drive,  WebDAV  will  authenticate  the 
user  using  the  CLE  authentication  function,  which  at  NPS  is  configured  to  use  the  NPS 
domain  authentication  services. 

Access  to  Sakai  resources  by  authenticated  users  is  controlled  by  a  Sakai 
application  policy  which  is  based  on  the  rights  assigned  by  the  instructor.  The  instructor 
would  selects  the  access  pennission  e.g.,  create,  read  or  delete  contents,  for  each  of  the 
folders  he  created,  as  shown  in  Figure  17. 
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Home 

Syllabus 

Lessons 

Resources 

Discussion  Forums 


Assignments 
Tests  &  Quizzes 


CyberCEIGE 

Z  Resources  9 

Permissions 

Set  permissions  for  resources  in  folder:  Summary  Report 

Clear  all  Undo  changes 


Announ  cements 

Permission 

Course  Develooerl  Guest  Lecturer! 

|  Instructs 

r|  Observer 

1  Student |Teachina  Assistant! 

Calendar 

content,  new 

✓ 

✓ 

✓ 

□ 

o 

✓ 

Messages 

content,  read 
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✓ 

/ 

Chat  Room 

D 

o 

Roster 

content,  revise. any 

Gradebook 

content.revise.own 

✓ 

✓ 

✓ 

n 
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✓ 
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content.delete.any 

□ 

□ 

✓ 

0 

Site  Editor 

Site  Statistics 

content. delete. own 

✓ 

✓ 

✓ 

ft 

D 

✓ 

Search 

content.hidden 

✓ 

✓ 

✓ 

Irl 

Irl 

✓ 

Help 

o 

1L_J 

Wei  Chean  Um 

[  Save  J  |  Cancel 

Figure  17. 

Permission  Assignment  in  CLE. 

This  thesis  employed  the  following  steps  to  upload  the  report  to  the  instructor: 

•  The  instructor  managing  the  CLE  course  site  adds  the  CyberCIEGE 
administrator  as  a  member  of  the  course. 

•  The  administrator  logs  into  the  course  site  and  obtains  the  URL  link  from 
the  resource  page  as  shown  in  Figure  18. 


CyberCEIGE 

z  Resources 


Upload  Down  load  Multiple  Resources 

The  instructions  on  this  page  show  you  how  to  create  a  folder  on  your  desktop  machine  that  will  allow  you  to  drag  anc 
site's  Sakai  Resources  tool. 

This  involves  using  a  protocol  called  WebDAV.  The  WebDAV  setup  process  is  different  for  each  operating  system.  Fol 
Mote:  if  you  simply  want  to  upload  files  to  the  Resources  tool  one  by  one,  dick  Site  Resources  at  the  top  of  the  pagt 
Step  1  -  Highlight  the  following  URL  and  cqpv  it  to  vcur  clipboard: 

https://cle.nps.edu/dav/7a4Sba65-07c6-44cf-9f95-d2S7b3c71ade 


:71ad^^ 


Upload-Download  Multiple  Resources 


Access  to  this  site's  resoarces  from  your  desktop 

The  foflowmz  instructions  will  help  you  set  up  access  to  this  site's  resources  folders  directly  from  your  desktop  so  you  can  a 


Figure  18.  CyberCIEGE  Resource  Page 
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•  The  administrator  will  then  map  the  URL  to  a  particular  drive  on  his 
system. 

•  Once  the  drive  is  mapped,  the  administrator  uploads  the  generated 
summary  report  by  indicating  the  path  on  the  Input  and  Output  Selector. 

In  the  event  the  instructor  is  not  comfortable  giving  the  administrator  rights  to 
insert  materials  into  the  course  site,  an  alternative  is  for  the  CyberCIEGE  administrator  to 
manage  a  “CyberCIEGE”  course  site  himself.  In  the  course  site,  the  administrator  will 
published  all  the  summary  reports  for  the  various  classes  onto  the  resource.  Once  that  is 
done,  the  instructor  can  either  go  into  the  CyberCIEGE  course  site  to  pull  out  the 
summary  report  or  he  can  create  a  link  in  his  course  site  for  easy  reference.  The  steps  for 
creating  the  link  are  as  follows: 

•  The  instructor  selects  “Edit  Tools”  option  in  the  “Site  Editor”  of  his  CLE 
course  site. 

•  Once  in  the  “Edit  Tools”  window,  the  instructor  selects  “Web  Content” 
which  allows  him  to  create  a  link  that  is  mapped  to  the  summary  report  as 
shown  in  Figure  19. 


£  Site  Editor  u 

Add  Multiple  Tool  Instances 

Add  multiple  tool  instances  or  configure  tool  options.  You  can  also  make  configuration  adjustments  later  in  the  site  using  the  O 
Web  Content 

Title  |  CyberCIEGE  ]  (Suggested  length  15  char.) 

Source  http:// 

<Q  More  Web  Content  Tools?  ’ 

|  Contmue~~ |  [  Back  ]  |  Cancel 


Figure  19.  CLE  Site  Editor  Feature. 


31 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


32 


V.  IMPLEMENTATION  AND  TESTING 


This  chapter  contains  two  sections.  The  first  section  describes  the  implementation 
of  the  functions  highlighted  in  Chapter  IV.  The  second  section  covers  the  test  strategy 
employed  for  the  verification  of  the  developed  functions. 

A.  IMPLEMENTATION 

Chapter  IV  discussed  the  development  strategies  for  four  key  improvements  to  the 
analysis  module,  summary  table  for  event  log  analyzer,  summary  report  generation 
function,  various  levels  of  summary  report  and  integration  with  CLE.  The 
implementation  of  these  improvements  utilized  the  current  CyberCIEGE  code,  written  in 
Java,  and  classes  as  much  as  possible.  The  developments  were  done  using  the  NetBeans 
Integrated  Development  Environment  (IDE)  7.0,  which  is  a  free  and  open  source  IDE. 

Some  of  these  developments  could  be  implemented  using  the  GUI  design  features 
of  NetBeans.  However,  it  was  not  done  so  because  the  existing  Campaign  Analyzer  code 
base  uses  manual  construction  of  window  components.  An  exception  is  the  new  class 
that  was  created  to  allow  the  user  to  select  the  input  files  and  output  directory  through  a 
dialog  window.  Also,  the  existing  directory  structures  are  not  consistent  with  NetBeans 
project  management  assumptions  and  thus  compilation  was  perfonned  outside  of  the  IDE 
using  manual  scripts. 

1.  Implementation  of  Improved  Event  Log  Analyzer 

The  “EventLogGUI”  class  was  modified  to  include  the  additional  features  for 
event  log  display.  Two  key  supporting  methods  were  created  for  the  inclusion  of  the  new 
features.  The  first  method  that  was  created  is  “makeSummaryPanel”.  This  method 
defines  the  layout  of  the  new  summary  panel  and  it  initiates  retrieval  of  the  selected  game 
details  from  the  log  file.  The  second  method  created  was 
“onSummaryTableValueChanged”.  This  method  an  active  listener  that  waits  for  the 
user’s  selection  on  the  summary  table  and  loads  the  selected  event  details  on  the  middle 
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panel.  A  “summaryData”  method  was  created  in  the  “PlayerStatus”  class  to  collect  the 
required  information  from  the  selected  game’s  log  file.  The  method  uses  the  game 
number  to  parse  the  logs  to  retrieve  the  summary  for  that  particular  game  session. 
Summary  of  the  modifications  is  as  listed: 

•  Included  a  summary  pane  on  the  top  of  the  event  log  analyzer  window. 

•  Function  to  pull  in  the  required  data. 

•  Display  events  based  on  the  selection  from  the  summary  pane. 

•  Highlight  and  display  the  event  log  for  the  last  game  by  default. 

•  Removed  the  display  game  feature. 

2.  Implementation  of  Summary  Report 

“CampaignAnalyzerGUI.java”  was  modified  to  create  the  summary  report 
feature.  A  new  class  “SumlnputGUI”  was  also  created  to  generate  an  input  window  for 
the  user  to  define  the  paths  for  the  required  files.  Once  the  user  defines  the  respective 
paths,  the  two  new  methods  “readLabFile”  and  “readStudentFile”  will  parse  the  files  to 
get  the  lab  mapping  and  student  information  respectively.  All  the  required  infonnation  is 
then  retrieved  from  the  logs  and  stored  in  a  multidimensional  array  and  eventually  passed 
to  “outhtml”  for  printing  out  the  HTML  summary  report.  The  HTML  is  embedded  in  the 
code.  Chapter  VI  identifies  future  work  to  make  the  HTML  output  configurable  by  the 
user.  The  modifications  are  as  listed: 

•  Created  the  generate  report  button. 

•  Included  the  function  to  read  the  input  files. 

•  Included  the  function  to  populate  data  for  the  summary  report. 

•  Included  a  function  to  generate  the  summary  report. 

B.  TEST  STRATEGY 

In  order  to  validate  the  developed  tools  worked  as  intended,  a  two-stage 
distributed  testing  procedure  was  used.  At  the  end  of  each  development  phases,  the 
product  will  be  first  tested  locally  with  the  sample  log  files  provided  by  the  CyberCIEGE 
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project  engineer.  The  second  stage  was  to  implement  the  developed  tools  in  the  actual 
system  and  test  it  with  actual  log  files  in  the  database  by  the  engineer. 

1.  Testing  of  Improved  Event  Log  Analyzer 

Verification  tests  were  conducted  to  test  whether  the  summary  table  displays  the 
correct  infonnation,  including  the  number  of  games  played  by  the  student.  Each  of  the 
games  displayed  on  the  summary  table  was  also  selected  individually  to  test  whether 
correct  information  was  displayed  on  the  Event  panel.  The  student’s  log  with  the  follow 
number  of  games  was  selected: 


No.  of  Games  to  be  Displayed 

Result  of  Test 

1  game 

Successful 

3  games 

Successful 

5  games 

Successful 

7  games 

Successful 

9  games 

Successful 

Table  2.  Event  Log  Analyzer  Test  Cases  and  Result 


The  summary  table  was  able  to  display  the  correct  infonnation  and  the  correct 
number  of  games  as  stated  in  each  of  the  selected  logs.  All  the  selected  games  were  also 
able  to  display  the  correct  event  log. 

2.  Testing  of  Summary  Report 

The  following  test  cases  were  created  to  verify  the  summary  report  function  could 
pull  the  required  information  based  on  the  input  given  by  the  user.  The  tests  were  set  to 
verify  the  following: 

•  The  function  is  able  to  read  the  infonnation  from  the  input  files  correctly. 

•  The  function  is  able  to  locate  the  conect  game  logs  from  the  various  game 
folders. 

•  The  function  is  able  to  collect  the  required  information  from  the  log  files. 
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•  The  function  is  able  to  generate  the  summary  report  in  the  correct  fonnat. 

•  The  function  is  able  to  output  the  summary  report  at  the  defined  location. 

In  order  to  verify  the  function  is  able  to  perform  the  abovementioned  features,  the 


function  was  tested  with  the  following  input: 


No.  of  Lab  to  be  Displayed 

Number  of  Student 

Result  of  Test 

1  lab 

10  students 

Successful 

2  labs 

20  students 

Successful 

3  labs 

30  students 

Successful 

4  labs 

40  students 

Successful 

Table  3.  Summary  Report  Test  Cases  and  Results 


The  function  is  able  to  generate  the  summary  reports  capturing  all  the  required 
information. 

3.  Testing  of  Integration  to  CLE 


A  CyberCIEGE  account  was  created  in  NPS  CLE.  This  account  was  used  to  test 
the  uploading  and  downloading  of  the  summary  reports.  The  results  of  the  tests  are  as 
follows: 


Test  Description 

Result  of  Test 

Mapping  of  drive  using  Window  7  system 

Successful 

Uploading  of  reports  to  the  resource  folder 

Successful 

Viewing  of  the  reports  by  another  user 

Successful 

Creating  link  on  another  course  site 

Successful 

Viewing  of  report  using  the  link 

Successful 

Uploading  of  report  from  Campaign  Analyzer 

Successful 

Table  4.  CLE  Test  Cases  and  Results. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

CyberCIEGE  is  designed  to  be  an  educational  instrument  to  enhance  the 
delivering  of  network  security  knowledge.  Therefore,  it  is  important  to  create  a  learning 
management  platform  so  that  an  instructor  is  able  to  assess  whether  the  learning 
objectives  are  being  met  and  to  monitor  the  progress  of  his  students. 

To  create  the  learning  management  platform,  we  first  looked  at  how  a  scenario 
can  be  better  designed  such  that  the  key  infonnation  and  lessons  are  delivered  in  the 
game  play.  This  led  to  analysis  of  how  key  information  can  be  captured  and  presented  in 
the  current  event  log  analyzer,  and  in  the  form  of  summary  reports  for  the  instructors  as 
described  in  Chapters  III  and  IV.  The  thesis  also  developed  a  process  whereby 
CyberCIEGE  can  be  linked  to  the  collaborative  learning  environment  which  allows  a 
smoother  transfer  and  management  of  the  summary  reports. 

During  the  course  of  developing  the  products  for  this  thesis,  some  other  potential 
enhancements  were  also  identified.  However,  due  to  the  limited  timeframe  for  this  thesis, 
these  enhancements  were  not  implemented.  These  additional  features  will  be  explained  in 
detail  in  the  next  section  for  possible  future  work  on  the  learning  management  platform. 

B.  RECOMMENDATIONS 

1.  Capturing  of  In-Game  Questions  Results 

The  CyberCIEGE  engine  was  modified  to  implement  the  in-game  questioning 
during  the  game  play.  These  in-game  questions  are  in  the  form  of  multiple  choice 
questions  that  can  keep  looping  till  the  correct  answer  is  selected.  The  improved 
summary  report  did  not  managed  to  capture  and  utilized  these  information  for  analysis.  It 
is  recommended  that  the  sequence  of  the  student’s  selection  could  be  captured  and 
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presented  in  the  summary  report.  This  information  may  help  the  instructor  to  assess  the 
student’s  level  of  understanding.  The  information  could  be  captured  in  the  level  2  report 
as  shown  in  Figure  19. 


STUDENT001 


LABI 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUIT) 

POINT  OF  LOST 

QUESTION 

(CORRECT  1ST  TIME/NOS 
OF  QNS) 

1 

01-Jul-ll 

3 

LOST 

Phase  1 

(0/1) 

2 

03-Jul-ll 

7 

WON 

— 

(2/3) 

LAB  2 


ATTEMPT  NOS 

DATE 

TIME  SPENT  (MIN) 

RESULT 

(WON/LOST/QUIT) 

POINT  OF  LOST 

QUESTION 

(CORRECT  1ST  TIME/NOS 
OF  QNS) 

1 

22-Jul-ll 

2 

QUIT 

.... 

(0/1) 

2 

22-Jul-ll 

5 

LOST 

Phase  2 

(1/2) 

3 

22-Jul-ll 

10 

WON 

— 

(2/2) 

Figure  20.  Insertion  of  the  question  information  in  the  level  2  summary  table. 


The  suggested  format  is  as  shown  in  the  last  column  of  Figure  19.  A  summary  of 
how  many  times  the  student  got  the  correct  answer  on  the  first  try  and  the  total  number  of 
questions  in  the  particular  scenario.  In  the  event  when  the  instructor  would  like  more 
information,  he  could  select  the  respective  cell  and  the  next  level  of  report  would  list  all 
the  questions  for  that  scenario  and  the  student’s  answer  sequence. 

2.  Use  of  Graphs  and  Charts  for  Summary 

As  highlighted  in  Chapter  III,  the  attempt  and  scenario  analysis  charts  and  graphs 
could  provide  better  analytical  information  for  the  instructor.  Therefore,  additional 
features  to  generate  these  charts  could  be  created  in  the  summary  report  or  the  charts 
could  be  pre-generated  and  included  into  the  summary  report. 

3.  Usage  of  GUI  Builder 

The  GUI  builder  function  in  the  NetBeans  IDE  provides  a  much  simpler  way  to 
create  tables  and  windows  similar  to  what  the  Campaign  Analyzer  and  Event  Log 
Analyzer  currently  implement.  It  would  be  useful  if  these  displays  were  recreated  with 
the  GUI  builder  to  enhance  future  expandability. 
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4. 


Enhanced  Integration  with  CLE 


The  ideal  integration  between  CyberCIEGE  and  CLE  is  to  allow  the  CyberCIEGE 
to  run  in  CLE  environment.  This  would  allow  the  instructor  to  post  the  relevant  lab  in  the 
course  site  in  CLE,  the  students  would  just  need  to  click  on  the  link  and  proceed  with  the 
lab  just  like  the  current  test  and  assessment  features  in  CLE.  At  the  end  of  the  lab  session, 
an  auto  generated  summary  report  will  be  created  in  the  CLE  site  for  better  management. 

However,  running  CyberCIEGE  within  the  CLE  environment  is  not  possible  as 
the  CyberCIEGE  GUI  utilizes  the  DirectX  based  3D  hardware  acceleration  that  is  not 
currently  available  in  web  browsers.  Additionally,  converting  CyberCIEGE  to  use  a  new 
set  of  graphical  interfaces  would  be  a  major  undertaking.  A  more  achievable  integration 
of  CyberCIEGE  with  the  CLE  might  be  to  utilize  SCROM  functions  to  define  campaigns 
and  to  guide  students  through  these  campaigns.  For  example,  the  CyberCIEGE 
Campaign  Player  could  retrieve  campaign  information  from  the  CLE  rather  than  from 
local  game  storage. 

5.  Configurable  Summary  Report  Formats 

Color-coding  is  used  in  the  summary  report  to  highlight  student’s  progress  e.g., 
red  color  for  a  not  attempted  game,  yellow  color  for  an  uncompleted  game.  This  set  of 
color-coding  is  embedded  in  the  code  and  it  would  be  difficult  for  user  to  change  the 
color  scheme.  Therefore,  a  Cascading  Style  Sheet  (CSS)  could  be  implemented  to 
provide  more  flexibility  for  the  user  to  change  the  presentation  scheme  e.g.,  color,  fonts, 
without  changing  the  code.  Control  to  the  report  fonnatting  e.g.  colors,  fonts. 

6.  Extended  Student  Assessment 

The  improved  summary  report  includes  of  two  levels  of  detail  for  the  instructor  to 
analyze  the  student’s  progress  in  each  of  the  scenarios.  However,  the  implementation 
would  still  require  the  instructor  to  refer  to  the  scenario  definition  or  separate  instructor 
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notes  to  understand  exactly  what  problem  the  student  is  facing.  Therefore,  a  third  level  of 
detail  could  be  implemented  to  allow  an  automated  generation  of  analysis  based  on 
proxies  that  are  built  in  the  game  scenario. 

The  implementation  of  the  third  level  of  detail  is  more  complicated  as  it  would 
require  the  scenario  designer  to  tag  an  analysis  field  for  each  question’s  answer,  phases 
and  objectives  in  the  game  scenario.  The  report  generator  would  then  pull  this 
information  depending  on  the  progress  of  the  student  and  generate  the  analysis  of 
possible  problems  that  a  particular  student  faced.  A  simple  illustration  for  this 
implementation  is  that  when  the  scenario  designer  implements  the  questions  in  the  game, 
he  will  tag  an  analysis  note  to  each  of  the  wrong  answer  that  the  student  chose.  Therefore, 
in  the  event  when  the  student  chose  the  wrong  answer,  the  summary  report  engine  will 
pull  the  analysis  note  out  indicating  that  the  student  might  not  understand  a  certain 
concept.  In  the  case  where  a  student  only  progressed  until  the  particular  phase  of  the 
game,  the  summary  report  will  also  reflects  the  possible  issues  faced  by  the  student  as 
indicated  in  the  analysis  note  created  by  the  scenario  designer. 

Although  the  implementation  would  promote  a  certain  level  of  in-game 
assessment  of  the  student’s  capability,  we  would  want  to  retain  the  initial  idea  of 
promoting  the  idea  for  student  to  explore  the  various  options  provided  by  the  game.  Thus, 
the  analysis  algorithm  could  adopt  the  logic  that  if  a  student  returns  to  play  the  scenario 
after  he  has  successfully  completed  the  scenario,  the  wrong  choices  that  he  made 
subsequently  would  not  be  reflected  in  the  summary. 

Allow  scenario  designer  or  instructor  to  tag  a  specific  game  objectives  and 
questions  to  specific  learning  objectives  such  that  an  instructor  can  obtain  an  automated 
indication  of  what  is  the  problem  faced  by  the  student. 
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